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Introduction 

The Job Training Partnership Act (JTPA) of 1983 Is the primary 
federally funded employment and training program In the United States 
(General Accounting Office, 1986). The program is administered by the U.S. 
Department of Labor and state governments. At the local level, JTPA 
programs are provided through 582 Service Delivery Areas (SDAs; General 
Accounting Office, 1987). The Houston Job Training Partnership Council 
(HJTPC) is one of three SDAs in the Houston area. In addition to federal 
and state authorities, the HJTPC is accountable to the Council of the City 
of Houston and a Private Industry Council (PIC) composed of individuals 
from a wide range of organizations and occupations. 

A major function of the JTPA is to provide job training for disadvan- 
taged youth. Recently, there has been an increasing emphasis in Congress 
on the importance of competency training in basic academic areas, as dis- 
tinct from job training and placement services, for youth participating in 
JTPA programs. Partly for this reason, and partly for purposes of accoun- 
tability, the HJTPC committed to evaluating the effectiveness of those 
programs providing training in academic competencies during summer 1986. 
The Research and Evaluation Department of the Houston Independent School 
District (HISD) agreed to conduct an evaluation of the summer academic 
programs for secondary level youth. 
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Methods 

The eight-week summer training programs were provided by six organiza- 
tions under contract with the HJTPC. Three of the programs were offered at 
more than one site; In all, there were fourteen sites. 

Data collection during the summer Involved pretesting and posttestlng 
at each of the sites on achievement In reading, mathematics, and writing; 
pretesting and posttestlng on self-esteem and self-ef f Icacy; and surveying 
participants, teachers, and program coordinators on their assessments of 
the programs. Subsequently, during the 1986-87 school year, participants 
were tracked on the following variables: course grades, course proficiency 
test scores, performance on the Texas Educational Assessment of Minimum 
Skills (TEAMS), school attendance, and dropout rates. Identical school- 
year data were collected on a comparison group of HISD students matched by 
school, grade, gender, and ethnicity. 

There were 1505 students In the sample of HJTPC participants and In the 
comparison group. Sample sizes varied for data analysis for a variety of 
reasons: a) missing data on specific variables, b) exclusion of one site 
from analysis of pretest/posttest achievement tests because of cheating by 
students on the posttests, c) exclusion of one program from analysis be- 
cause of the small number who were pretested and posttested, and because 
school-year data were not available, d) difficulty In locating students In 
the school district's computerized masterflle because of missing or 
Incorrect Identification numbers, and e) absence of either pretest or post- 
test data from the summer, which Indicated that the student may not have 
attended the program for the entire eight weeks. 
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The following discussion will address summer pretest/posttest data, 
student and staff assessment data, and the school-year data separately. 

Pretest and Posttest Data from Summer 1986 

Measures 

Pretest measures were the Diagnostic Tests for TEAMS In reading, math» 
and writing, that previously had been constructed by the testing department 
In HISD to assist teachers In preparing students to take the TEAMS. The 
posttest measures were constructed by the testing department specifically 
for the HJTPC evaluation. Both sets of achievement measures were con- 
structed to match the objectives covered on the TEAMS. Iiese objectives are 
listed In the Appendix. 

The Rosenberg Self-Esteem Scale (Rosenberg, 1965) was used to measure 
global self-regard at the beginning and end of the summer. To measure 
self-ef f Icacy (Bandura, 1982; Shunk, 1985), Items were constructed asking 
participants their expectations about school. They reported whether they 
thought they would be able to get along well with other students, attend 
school unless sick, get along with teachers, get a job, stay In school 
until graduation, do the work In school, get good grades, make friends, and 
learn what they need to get a job. 

Results 

Statistical analyses were based on matched samples of cases. Students 
were Included who had taken both pretest and posttest on the measure being 
analyzed, and for whom student Identification numbers could be located In 
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the HISD student masterflle. Analyses addressed whether the various 
programs and sites varied significantly from each other on the measures of 
achievement, self-esteem, and self-ef f Icacy. 

Two techniques were used to compare among programs and sites on the 
achievement measures. In the first, all pretest and posttest scores were 
converted to Z scores, and gains In Z scores were calculated from pretest 
to posttest. Analysis of variance was then used to calculate differences 
among the programs and sites on achl lavement • The second technique was ana- 
lysis of covarlance, which allowed for a comparison among the posttest 
scores of programs and sites after the scores had been adjusted for dif- 
ferences on the pretest. 

The left side of Table 1 summarizes the outcomes of the analysis of 
variance on gains In Z scores, and the right side shows outcomes of the 
analysis of covarlance. Average percents of Items correct on each test at 
pretest and posttest are also shown. 

Average scores on the measures of self-esteem and self-ef f Icacy are 
shown In Table 2. Analysis of variance revealed no significant differences 
among programs or from pretest to posttest. 

Conclusions 

The standardized gain score and ANCOVA analyses did Indicate statisti- 
cal differences among; the programs and sites, and the differences tended to 
be consistent across the two methods of analysis. However, the success of 



Table 1 



Comparison of Pretest with Post test Achievement 
Among Programs and Sites 



Analysis of Variance on 
Standardized Gain Scores 



Program 



ELEC 

SAS 

SKY 

CIS 

CT 

Site 



Mean 

Standardized Standard N 



Gain 
.20^ 



Deviation 



Reading 

Analysis of Covariance on 
Number of Items Correct 



Program 



Adj us ted 
Mean on 
Posttest 



.77 


97 


ELEC 


26. 98^ 


.83 


169 


SAS 


26.26 


.79 


144 


SKY 


26. lib 


.93 


146 


CIS 


24. 14c 


.82 


211 


CT 


23. 60^ 



Observed 
Mean on 
Posttest 

26.39 
26. 24 
27.01 
22.73 
24.25 



mean 
Percent 
Items 

Correct on 
Pretest 

56 
59 
63 
52 
62 



Mean 

Percent 

Items 

Correct on 
Posttest 

78 
77 
79 
67 
71 



97 
166 
143 
146 
211 



00 
ID 



SAS Mt. Sinai 


.25 


ELEC 


.20^ 


SAS Gethsemanie 


.11 


SAS Imiac. Cone. 2 


.08 ■ 


CIS Dowling 


.02 


SAS Mt. Corinth 


-.05 


SKY 


-.07 


SAS Resurrection 


-. 13 


SAS Imoac. Cone. 1 


-. 15 


CIS Ryan 


-. 26 


CT AM 


-. 36*' 


CT PM 


-.37 



.74 


13 


SAS Mt. Sinai 


27.02 


26.23 


55 


.77 


97 


ELEC 


26. 89^ 


26.39 


56 


.83 


28 


SAS Mt. Corinth 


26.75 


26.89 


59 


.67 


31 


SAS Immac. Cone. 2 


26.30 


26.26 


58 


1.02 


66 


SKY 


26.00 


27.01 


63 


.99 


38 


SAS Immac. Cone. 1 


25.99 


28.21 


69 


.79 


144 


SAS Gethsemanie 


25.99 


25. 12 


54 


.57 


30 


SAS Resurrection 


24.74 


24.57 


58 


.65 


24 


CIS Dowling 


24.42b 


21.83 


46 


.83 


80 


CT AM 


23.82C 


24.82 


63 


.71 


113 


CIS Ryan 


23.76 


23. 46 


57 


1.01 


63 


CT PM 


22.93^ 


22.43 


56 



77 
78 
79 
77 
79 
83 
74 
72 
64 
73 
69 
66 



13 
97 
36 
31 

143 
24 
28 
30 
66 

113 
80 
63 
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Table 1 (continued) 



Analysis of Variance on 
Standardized Gain Scores 



Meaii 

Program Standardized Standard N 

Gain Deviation 



ELEC 


.00^ 


.51 


94 


CIS 


-. 00^ 


.90 


136 


SAS 


-. 15 


.72 


174 


SKY 


-. 20 


.34 


148 


CT 


-. 25*' 


.49 


221 


Site 








CIS Oowling 


.23^ 


.93 


58 


SAS Mt. Sinai 


.19 


.61 


13 


SAS Mt. Corinth 


.08 


.84 


40 


ELEC 


.00 


.51 


94 


SAS lomac. Cone. 1 


-.15 


.47 


24 


CIS Ryan 


-.17 


.84 


78 


SAS Gethsemanie 


-.18 


.87 


29 


SKY 


-.20 


.54 


148 


SAS Resurrection 


-.22 


.45 


29 


CT AM 


-.23 


.46 


113 


SAS Immac. Cone. 2 


-. 25 


.49 


33 


CT PM 


-.34 b 


.55 


73 



Math 

Analysis of Covariance on 
Number of Items Correct 

I 



Ad j us ted Obse rved 

Program Mean on Mean on 

Post test Post test 

ELEC 39.09^ 39.03 

SAS 38.62 40.67 

CIS 38.60 33.99 

SKY 37.42 37.75 

CT 36.64b 37.64 



*^ 

OQ 

I (D 

[Mean Mean ^ 

Percent Percent 

Items Items 

Correct on Correct on N 

Pretest Posttest 

66 75 94 
72 78 171 
55 65 133 

67 73 145 
69 72 221 



SAS Mt. Sinai 


41.14 


41.69 


68 


80 


13 


CIS Dowling 


40. 92^ 


34.32 


50 


66 


56 


SAS Mt. Corinth 


40.84 


40.41 


65 


78 


39 


ELEC 


38. 96^ 


39.03 


66 


75 


94 


SAS Immac. Cone. 1 


38.47 


43.33 


78 


83 


24 


SAS Resurrection 


37.43b 


41.14 


75 


79 


29 


SAS Immac. Cone. 2 


37.31 


41.79 


77 


80 


33 


SKY 


37.28 


37.75 


67 


73 


145 


SAS Gethsemanie 


36.96 


37. 10 


66 


71 


29 


CIS Ryan 


36. 81^ 


33.74 


59 


65 


77 


CT AM 


36.67C 


37.46 


68 


72 


113 


CT PM 


35.49c 


36.51 


69 


70 


73 
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Table 1 (continued) 



Writing 



Analysis of Variance on 
Standardized Gain Scores 



Analysis of Covariance on 
Number of Items Correct 



Program 



SKY 




ELEC 


-.03 


SAS 


07 


01 


13" 




o o n 
-. 23" 


Site 




SKY 


.19^ 


SAS Mt. Corinth 


.16 


SAS Immac. Cone. 


2 .06 


SAS Mt. Sinai 


.05 


CIS Dowling 


-.01 


ELEC 


-.03 


SAS Immac. Cone. 


1 -.06 


CT PM 


-.15 


CT AM 


-.15 


SAS Gethsemanie 


-.23 


SAS Resurrection 


-.39 


CIS Ryan 





Mean 

Standardized Standard 
Gain Deviation 

.74 
.80 
.77 
.84 
.89 



1 


1 






jMean 


Mean 












Percent 


Percent 




N 




Adjusted 


Observed 


Items 


Items 




Program 


Mean on 


Mean on 


Correct on 


Correct on 


N 






Posttest 


Posttest 


Pretest 


Posttest 




140 


SKY 


24. 32^ 


23.80 


47 


53 


140 


93 


SAS 


23.32b 


24.25 


52 


54 


172 


172 


ELEC 


22.97c 


22.81 


48 


51 


93 


229 


cr 


22.52C 


22.66 


49 


50 


229 


158 


CIS 


21.37c 


20. 7x 


46 


46 


158 



.74 


140 


SAS Mt. Corinth 


24. 63^ 


25.03 


50 


.68 


37 


SAS Immac. Cone. 1 


24.34 


27.00 


57 


.69 


33 


SKY 


24.33b 


23.80 


47 


.78 


14 


SAS Immac. Cone. 2 


23.73 


23.76 


49 


.88 


70 


SAS Mt. Sinai 


23.55 


23.36 


48 


.80 


93 


ELEC 


22.97c 


22.81 


48 


.74 


27 


CIS Dowling 


22.62C 


21.46 


45 


.78 


78 


CT AM 


22.61C 


23.22 


51 


.90 


116 


SAS Gethsemanie 


22.39 


23.64 


53 


.78 


28 


CT PM 


22.06C 


21.62 


47 


.88 


29 


SAS Resurrection 


21.39 


22.72 


53 


.86 


88 


CIS Ryan 


20.39c 


20.11 


48 



56 

60 

53 

53 

52 

51 

48 

52 

52* 

48 

51* 

45^ 



37 
27 

140 
33 
14 
93 
70 

116 
28 
78 
29 
88 



* Posttest is lower than pretest. 

Note: Means with superscripts (e.g., a, be) achieved statistical significance at the 0.5 level 
or lower. Means that do not share any superscript letters are significantly different. 
Those that share a superscript letter are not significantly different. Mean gain scores 
were compared with the Scheffe multiple range test. 
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Table 2 



Comparison of Pretest 
With Posttest on Self-Esteem 
and Self-Efficacy 





Pretest 


-Esteem^^^ . 
Posttest 


Self- 
Pretest 


Efficacy^ 
Posttest 


Computech 


38.51 


39.00 


13.20 


13.17 


ELEC 


39.16 


39.90 


12.66 


12.90 


CIS 


37.13 


36.96 


13.16 


12.62 


SKY 


41.12 


41.12 


12.61 


12.21 


SAS 


36.83 


38.22 


14.39 


14.64 



Scores may range from 10 to 50; a higher score indicates higher self- 
esteem and an increase from pretest to posttest indicates increased 
self-esteem. 



b Scores may range from 9 to 45; a lower score indicates higher self- 
efficacy and a decrease from pretest to posttest indicates increased 
self-efficacy. 
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programs and sites varied according to the academic content area tested, so 
that it is difficult to conclude that any one program or site was clearly 
outstanding. Overall, greater gains were made in reading and mathematics 
than in writing. This lower performance in writing is consistent with 
other studies of TEAMS scores done In HISD. The pretests and posttests 
were different, and there is no information available on their comparability. 
Nevertheless, the mean percent of items correct did improve from pretest 
to posttest, particularly in reading and mathematics (Table 1). 

There were broad variations in achievement among sites within each 
of the three programs that had multiple sites, suggesting that variables 
related to site were important in determining performance. That is, 
the quality of the local management and teaching staff, the comfort of 
the physical setting, and the quality of implementation at a particular 
site may have had more to do with the achievement of participants than the 
use of common ma^-erials or curricula at all sites delivered by a par- 
ticular of a program. 

There were no consistent differences among programs or gains from pre- 
test to posttest on the measures of self-esteem and self-efficacy. However, 
the measures used may not have been sufficiently sensitive to change oc- 
curing during a relatively short eight-week program. 

Staff and Student Assessments 

Methods 

A total of 1182 participants and ^7 teachers completed assessment sur- 
veys. Their responses and the responses of four program coordinators are 
summarized in the text below. 
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Results 

Although there wc^e variations among programs, between two-thirds and 
three-fourths of participating students found their teachers helpful, 
believed they had learned a lot, found the level of difficulty about right 
for them, and rated their programs well overall. About one-half rated their 
programs as well organized, while the other half believed the programs had 
some problems with organization. About one-half liked the programs better 
than regular school. Many found the facilities uncomfortable. 

Overall, teachers reported problems with program administration and 
organization, student discipline, and some aspects of the academic program. 
They felt their accomplishments to be in the areas of professional growth, 
and the personal, social, and academic growth of students. Teachers 
reported receiving support from their program coordinators and staff, but 
many found problems in the organization of the programs. About one-half 
were satisfied with their facilities, the instructional materials provided 
in their programs, and the services received from HJTPC, Almost 20 percent 
found the services from HJTPC poor, their students weak in ability, their 
facilities poor, and their programs poorly organized. 

Overall, program coordinators perceived their accomplishments to be 
helping students with basic skills, giving them some job experience, and 
assisting in their personal growth. Their major problems were with student 
discipline, identification of students certified for the program, delays in 
receipt of administrative materials from HJTPC, disorganization, and diffi- 
culties in scheduling the posttests. Problems with HJTPC dealt with fre- 
quent changes in guidelines, and delays in receipt of materials and ser- 
vices. Coordinators had some difficulty understanding and completing the 
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paperwork for HJTPC, One coordlnatc^r cited problems with students who did 
not speak English and others who required special education services. In 
general, coordinators recommended changes that have to do with program ad- 
ministration on the part of HJTPC, 

Recommendations 

Based on the survey information, several recommendations were made to 
HJTPC. First, program providers should be required to demonstrate that 
their facilities are adequate for the purposes of the program* Each provi- 
der "s proposal needs to indicate a clear behavioral management system for 
dealing with discipJinary problems, and an assurance that teachers will be 
hired who have experience working with students with discipline problems. 
The HJTPC should streamline administrative paperwork required from the 
programs. A longer time for planning is needed for each program. This 
time should be used to coordinate the services provided by HJTPC and each 
contractor, to establish clear administrative and organizational guide- 
lines, and to design and integrate program evaluation activities. There 
must be a coordinator for each program and site who is actively involved in 
the program and present at the site. Clear lines of communication must be 
defined within programs. The on-site coordinators should be responsible 
for program implementation, communication at the site, and with the program 
coordinator or HJTPC, implementation of guidelines, and implementation of 
all necessary changes. Finally, HJTPC should consider establishing a 
system for receiving and reviewing the complaints of teachers and students. 
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1986-87 School Year FollowUp 

Methods 

During the 1986-87 school year, participants were tracked on the 
following variables: 

1. course grades by semester for high school students, and for the 
year for middle school students, 

2. course proficiency test scores on HISD*3 High School Proficiency 
Tests (HSPTs), 

3. performance in the Texas Educational Assessment of Minimum Skills 
(TEAMS), 

4. school attendance, 

5. dropout rates as reported on school records. 

Identical data were collected on a comparison group of students 
matched by school, grade, sex, and ethnicity. There were 1128 students in 
both the HJTPC sample and the matched sample of HISD students. Analyses 
compared HJTPC participants with students in the comparison group. In 
addition, they compared among the five summer programs. 

Measures 

TEAMS. The Texas Educational Assessment of Minimum Skills is taken by 
students in Texas in every other grade, starting in first grade. Students 
must pass grade 11 TEAMS in order to graduate. The grade 11 or exit-level 
TEAMS may be taken several times in grades 11 and 12. The exit-level test 
covers language arts and mathematics, and the lower level tests cover read- 
ing, writing, and mathematics. In order to pass, a student must pass all 
content-area subtests on the TEAMS for that year. HJTPC participants 
in grades 7, 9, and 11 (or 12 if they had failed) took the TEAMS. 
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Grades, Middle school students (grades 6-8) receive final grades at 
the end of the school year, and high school students (9-12) receive final 
semester grades • A grade of 70 is required for passing a course. 

Proficiency Tests. The High School Proficiency Tests (HSPTs) were 
constructed by the testing department of HISD. These are criterion-refer- 
enced tests, matched to the objectives of specific high school courses. 
During the 1986-87 school year high school students took the HSPTs along 
with class finals, and test scores were averaged into the semester final 
course grades. 

Attendance. School attendance is calculated for each student as the 
percent of days in attendance out of the number of days enrolled. 

Dropouts. Dropout rates are very difficult to calculate in large 
school districts because of student mobility and associated difficulties in 
keeping accurate records. In HISD, campus attendance clerks record a stu- 
dent as a dropout if they have sufficient information that the student will 
not be returning to school. These reported dropouts are probably lower than 
the genuine dropout rate, but they were the only dropout data available on 
the computerized student masterfile for 1986-87. 

Results 

A series of t-tests on achievement variables from the preceding school 
year was done to determine if the HJTPC and comparison samples were compa- 
rable. Analysis of 1985-86 baseline data revealed no differences between 
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HJTPC and the comparison group on high school spring-semester grades in 
language arts or mathematics. However, there were significant differences 
between the middle-school samples from HJTPC and the comparson group on 
grades. The mean middle-school grades in language arts were 74.91 for 
HJTPC and 76.94 for the comparison group (t - 2.96; df - 863; p < .01). 
The mean middle-school grades in mathematics were 72.56 for the HJTPC and 
74.24 for the comparison group (t - -2.27; df = 933; p < .05). Because of 
these differences at the middle school levels, analysis of covariance was 
used to compare HJTPC with the comparison group on course grades. T-tests 
on the percent of items correct on the HSPTs were used to compare the 
1985-86 scores cf the HJTPC and comparison groups of high school students. 
The groups did not differ on these baseline scores in language arts or 
mathematics. 

The results which follow compare HJTPC participants with students in 
the comparison group on the school-year data. They also compare among the 
five summer programs. 

TEAMS. Table 3 shows the performance of students on each subtest of 
the TEAMS. On the exit-level test, higher percentages of the HJTPC partici- 
pants passed each subtest than the comparison group students, but the per- 
cents passing do not appear to be meaningfully higher for grades 7 and 9. 

An analysis of covariance was conducted to compare the performance of 
HJTPC participants and the comparison group on total scores in each content 
area. The TEAMS scores were adjusted on HSPT scores for high school stu- 
dents and on the composite percentile score of the Iowa Test of Basic 
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Skills (ITBS) for middle school students. There were no significant differ- 
ences between HJTPC and the comparison group at any grade level (Table 3). 

Table 4 summarizes the TEAMS mastery rates for the HJTPC and com- 
parison groups in terms of the number of students who passed all subtests 
on the TEAMS. The data show a significantly higher passing rate by the 
HJTPC participants on the exit-level test, but no meaningful differences 
between the two groups on the TEAMS for grades 7 and 9, 

It is concluded that taking part in the HJTPC programs was associated 
with higher mastery rates on the TEAMS for students who took the exit-level 
test. Nevertheless, when adjusted scores on the TEAMS subtests are con- 
sidered, HJTPC and the comparison group were not different at any grade 
level. 
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Table 3 

Performance of HJTPC Participants and the 
Comparison Group on the Texas Educational Assessment 
of Minimum SMlls (TEAMS) 

Grade 11 (Exit Uvel) 



HJTPC 



Comparison 



Percent Passing 
Language Arts 
Mathematics 



Total Scores 

Language Arts ^ 
Mathematics ^ 



95. IZ 
94. OZ 



58.1 
54.5 



(Adjusted)* 

(58.31) 
(58.59) 



78. 8Z 
76. IZ 



55.4 
47.9 



(Adjusted)* 

(57.27) 
(56.85) 



Sample Size 
Language Arts 
Mathematics 



81 
83 



(71) 
(48) 



66 
67 



(45) 
(23) 



Grades 7 and 9 



Percent Passing 
Reading 
Writing 
Mathematics 



HJTPC 



65. OZ 
53. IZ 
70. 7Z 



Comparison 



64. 9Z 
51. 4Z 
70. 9Z 



Total Scores 
Reading ^ 
Writing d 
Mathematics 



(Adjusted)** 

31.0 (32.32) 31.0 

19.1 (19.79) 19.1 
29.6 (31.54) 30.3 



(Adjusted)** 

(31.73) 
(19.47) 
(31.07) 



Sample Size 
Reading 
Writing 
Mathematics 



412 
405 
410 



(290) 
(286) 
(288) 



* total items » 72; passing score ■ 50 
^ total items = 72; passing score " 39 
grade 7: total items ■ 40; passing score 
grade 9; total items ■ 44; passing score 



26 
30 



367 
358 
361 



° total items 
^ total items 



24; passing score 
44; passing score 



16 (grade 7), 19 (grade 9) 
26 



(203) 
(199) 
(202) 



* Scores adjusted using ANCOVA on percent correct on HSPT 
** Scores adjusted using ANCOVA on ITBS composite percentile score 
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Table 4 

Mastery Rates on the Texas Educational Assessment 
of Minimum Skills (TEAMS) 



Grades 7 and 9 



Number 

Percent Achieving 
Mastery 



Grade 11 

HJTPC Comparison HJTPC Comparison 

73 42 156 144 

87. 6Z * 60. OZ 36.9% 38.7% 



Note: To achieve mastery, a student must pass all subtests on the TEAMS: 
subtests for grades 7 and 9 are Reading, Writing, Mathematics; sub- 
tests for grades 11 or 12 Language Arts and Mathematics. 

* Total percent for HJTPC grades 11 or 12 significantly higher than other 
total percents: Chi square = 4.19; df ~ 1; p < .05. 



Grades. The final 1986-87 grades for middle school students in the 
HJTPC and comparison groups were adjusted on grades from the previous year. 
Analysis of covariance showed no differences between the groups in the 
adjusted final grades in mathematics, and a small but significant dif- 
ference favoring the comparison group in language arts (Table 5). 

The two high school groups were comparable according to baseline data, 
so analysis of variance was used for analysis of course grades. The fall 
1986 and spring 1987 grades in language arts and mathematics were compared. 
No significant differences were found (Table 6). 
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Table 5 

Final Adjusted* Course Grades of Middle School Students 



Language Arts 



HJTPC Comparison 



Average Grade 73.78 75.39** 

Sample Size 229 184 

Mathematics 

Average Grade 74.06 73.83 

Sample Size 253 200 

* Grades were adjusted using ANCOVA to control for Initial difference bet- 
ween HJTPC and the comparison group. Covariates were grades from 
1985-86 in the same content area. 

** Comparison group was higher: F = 4.20; df = 1; p < .05. 



Table 6 

Final Course Grades of High School Students 



Mathematics 



Fall 1986 
HJTPC Comparison 



Spring 1987 
HJTPC Comparison 



Language Arts 

Average Grade 
Sample Size 
Standard Deviation 



75.47 
t'19 
9.92 



74.39 
464 
10.73 



75.22 

559 
10.20 



74.98 
453 
10.98 



Average Grade 
Sample Size 
Standard Deviation 



74.54 
592 
10.77 



73.48 
483 
11.88 



74.02 

558 
11.80 



73.47 
473 
11.67 
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Proficiency Tests. Once it was determined that HJTPC and the compari- 
son group were comparable on baseline data, t-tests were used for analysis. 
Xhere were no statistically significant differences between HJTPC and the 
comparison group in fall or spring semesters on proficiency tests in 
language arts or mathematics. Table 7 summarizes these outcomes. 



Table 7 

Performance on the High School 
Proficiency Tests (HSPTs) 








Fall 

HJTPC 


1986 
Comparison 


Spring 1987 
HJTPC Comparison 


Language Arts 

Percent of Items Correct 
Sample Size 
Standard Deviation 


67.01 
536 
13.74 


65.74 
402 
15.47 


67.42 

541 
14.24 


68.52 
447 
14.96 


Mathematics 










Percent of Items Correct 
Sample Size 
Standard Deviation 


59.41 
444 
19.66 


60.14 

339 
20.09 


60.15 
492 
17.78 


58.21 
419 
17.95 



Attendance. Overall, HJTPC showed a significantly higher attendance 
rate than the comparison group (Table 8). This outcome should be inter- 
preted with some caution since the effect size is small: the overall 
difference is only about two days. Secondly, the outcome can be attributed 
partly to the low attendance rate of the SKV comparison sample. Despite 
this cautious interpretation, the HJTPC programs, except for CIS, were con- 
sistently higher in school attendance than their comparison groups. 



24 



Page 20 



Table 8 

Rates of 1986-87 School Attendance for 
All Groups and Overall 





HJTPC 


Comparison 


Computech 


91.0% 


90.6% 


Sample Size 


252 


238 


ELEC 


91.9% 


90.0% 


Sample Size 


95 


100 


SKY 


91.9% 


87.0% 


Sample Size 


162 


153 


CIS 


91.3% 


91.7% 


Sample Size 


234 


231 


SAS 


91.2% 


90.5% 


Sample Size 


260 


266 


Total 


91.4% * 


90.2% 


Sample Size 


1003 


988 


Standard Deviation 


11.43 


12.90 



* HJTPC was significantly higher than the comparison group in attendance: 
F = 4.31; df = 1; p < .05. There were no statistical differences among 
HJTPC programs. 



Dropouts. A total of 28 students or 2.5 percent of the HJTPC school- 
year sample were recorded as dropouts in 1986-87. By comparison, 67 stu- 
dents or 5.9 percent of the comparison school-year sample were recorded as 
dropouts. Even though this may be a meaningful outcome, the size of the 
effect is low that it is difficult to interpret. 



25 



Page 21 



Conclusions 

Among students In grades 11 and 12 who took fhe exlt-level TEAMS, 
mastery rates were higher than In the comparison group • This difference 
between the groups on TEAMS performance did not occur for students In 
grades seven or nine, nor did It occur when the test scores were adjusted 
on previous test performance. There was a small but significant effect 
favoring HJTPC on school attendance. There were also fewer dropouts among 
the HJTPC participants. These outcomes suggest that participation In the 
HJTPC program may have been associated with attending and staying In 
school. Alternatively, HJTPC participants may have been motivated to 
attend school, as evidenced by their completing the summer program. 

There were no consistent differences among the five programs on the 
school-year data, suggesting that the programs used similar Instructional 
techniques. It appears that they were all about the same: classrooms with 
teachers hired for the summer, teaching basic skills In seemingly conven- 
tlona? ways. 

Conclusions and Discussion 

Outcomes of the Evaluation 

An essential evaluation question for this project is whether the HJTPC 
summer programs were successful in improving academic skills and school 
attendance among participants. There were positive effects on these 
variables, but the outcomes were not consistent. From pretest to posttest 
during the summer, participants gained in the percent of items correct on 
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tests In reading and. mathematics. However, the tests were not known to be 
comparable in dlflculty or content, so that real gains in achievement are 
not clear. During the school year, participants in grades 11 and 12 showed 
significantly greater mastery, defined as passing all subtests, on the 
state TEAMS than students in the comparison group. However, the partici- 
pants were not significantly more successful than comparison students on 
the subtests of the TEAMS when scoros had been adjusted on other school 
performance variables • HJTPC participants did attend school significantly 
more days during the school year following the summer program. Neverthe- 
less , the effect size of two days that produced this significant outcome 
is very small. Finally, over twice as many comparison students were 
recorded as school dropouts than participants during the 1 36-87 school 
year. At the same time, the number of recorded dropouts was small in both 
cases and the difference between the ^roups represents only about 3.5 per- 
centage points. 

A plausible alternative explanation for the school-year outcomes is 
that the participant and comparison groups were different, despite careful 
matching on appropriate variables. Participants were unique in that they 
were willing and motivated to attend summer school for eight weeks. Of 
course, they were paid minimum wage to attend, but it is likely that they 
were more motivated than their peers to achieve success given that they 
sought out and applied for the programs and attended the sometimes hot and 
uncomfortable facilities. 

For these reasons, and with major one exception, the conclusion of 
the study is that the HJTPC summer programs were not clearly successful in 
producing effects on the variables measured. It may be, of course, that 
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the programs will have a long-term and positive Influence on participants 
when they come to decision points about completing school or seeking 
employment • 

One Important exception to the conclusion stated above has to do ith 
the performance of exit-level 11th and l?th grade students on the TEAMS. 
Their mastery levels were meaningfully higher than those of comparison 
students, and It Is concluded that the HJTPC programs were effective In 
helping these stuJents pass the TEAMS. There are several reasons for 
this positive conclusion. First of all, scores of these students on the 
TEAMS subtests were higher, even though they did not reach significance 
In the analysis of covarlance. (Table 3). The summer programs did assist 
them to gain the additional skills needed to pass the TEAMS and achieve a 
significant step forward graduation. A second reason for the positive 
conclusion Is that previous evaluations of other summer TEAMS preparation 
programs for high school students In HISD have shown positive outcomes. 
Programs providing training for the TEAMS could be expected to be success- 
ful. Finally It appears that the HJTPC summer programs were particularly 
appropriate and helpful for 11th and 12th grade students who were motivated 
to stu^v during the summer to pass the TEAMS and go on to graduation. 
It is not known if HJTPC created motivation for success in students, but it 
appears to have taken good advantage of the motivation that already 
existed. 
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To summarize, the general conclusion from the evaluation is that the 
HJTPC programs were not particularly effective in improving the performance 
or attendance of participating students. The exception is that uppor-level 
high school students, who were probably motivated to complete academic 
requirements for graduation, were helped by the programs. 

Thp evaluation described included observations of the summer classes 
during posttesting only. Process data were composed of staff and student 
assessments. Though limited, this information does give an idea of the 
quality of the summer programs. Assessments indicated a need for more 
thorough planning, coordination, and administration of programs by the 
providers and HJTPC. Brief on-site observations revealed apparently stan- 
dard approaches to curriculum and instruction. In many cases, the 
instructors were teachers hired for summer work. There was no evidence of 
special training for teachers or special curricula to address the require- 
ments of remedial education for disadvantaged and low-performing students. 
In other words, the summer programs did not appear to be particularly 
innovative. Obviously, any definite conclusions about the quality of the 
programs can not be made, but there was an indication that the programs 
could have been administered more effectively, and could have addressed 
more carefully the needs of remedial students. 

Utilization of the Evaluation 

A major issue in the evaluation of the HJTPC summer programs was the 
purpose of the research. The research was designed to determine program 
effectiveness in terms of student performance and school attendance, and 
provide information to HJTPC on student and staff assessments. It was 
intended to compare the various program providers and assist HJTPC in 
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making decisions about subsequent contracts to program providers • As it 
turned out, there were other expectations by HJTPC and program providers, 
and some of these expectations were not met# For example, some of the 
program providers requested information on the TEAMS performance of 
individual students* Such data are confidential and could not be provided. 
Some of the contractors were to be paid additional money depending on the 
number of their participants who passed TEAMS* However, given our best 
effort, the data still contained missing data on some of the participants* 
HJTPC wanted to use evaluation outcomes for public relations purposes, but 
the data were not sufficiently clear-cut for that purpose* The format of 
the reports to HJTPC was too technical in some cases; data needed to be 
prepared so that members of the Houston City Council and representatives 
of private industry could grasp its meaning easily* A simple presentation 
of complex data was difficult without belying the meaning of the outcomes* 
Finally, the two public agencies, HJTPC and the school district, have 
internal organizational procedures and requirements that made it very 
difficult to achieve flexible, responsive, and efficient approaches to 
problems in the evaluation as they arose* 

Despite these problems, the importance of this kind of evaluation 
remains* JTPA provides a great deal of money for basic skills training, 
and program effectiveness must be measured* Our society's disadvantaged 
depend on receiving training like that provided by JTPA to assist them in 
finding and maintaining jobs* Ideally, evaluation will contribute to the 
success of such {)ro^rams* 
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Objectives Included in the 
Pretests and Posttests 



Reading Objectives 

!• Determine sequence of events 

2. Follow written directions 

3« Use parts of a book as aids in locating information 

4« Use books as aids in locating Information 

5. Use graphic sources to get Information 

6« Use context to understand the meaning of words 

7« Use structure of a word to derive meaning 

8. Recognize and Interpret figurative language 

9* Identify main Idea 

10. Develop generall'^atlons from details 

11. Recognize like and unlike relationships 

12. Perceive cause-effect relationships 

13. Draw logical conclusions 

14. Predict future actions or outcomes 

l5« Explain and relate to feelings of characters 

16. Make judgments on the basis of Information 

17. Distinguish between fact and opinion 



Math Objectives 

1« Add whole numbers 

2« Subtract whole numbers 

3. Multiply whole numbers 

4« Divide whole numbers 

5« Charts and graphs 

6« Averages 

7« Compare fractions am' mixed numbers 

8« Add fractions and mixed numbers 

9« Subtract fractions and mixed numbers 

10. Multiply fractions and mixed numbers 

11« Divide fractions and mixed numbers 

12. Compare two numbers 

13« Round decimals and whole numbers 

14« Add decimals/word problems 

15« Subtract decimals/word problems 

16. Multiply declmals/worJ problems 

17. Divide decimals/word problems 

18. Identify equivalents 

19. Solve money problems 

20. Find a given percent 
21 • Find the base or rate 
22. Solve the equation 

23* Standard numerals and powers 

24. Choose metric unit 

25. Convert units of measure 

26. Scale drawings/Map reading 
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Appendix (cont.) 



Writing Objectives 

1. Demonstrate knowledge of capitalization 

2. Demonstrate knowledge of punctuation 

3. Recognize the correct spelling 

4. Demonstrate knowledge of correct English usage 

5. Demonstrate the ability to recognize fragments and/or run-ons 
6« Recognize sentence that best combines two related sentences 
7. Demonstrate the ability to proofread 
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